Whose emotion matters? Speaking activity localisation without prior knowledge

The task of emotion recognition in conversations (ERC) benefits from the availability multiple modalities, as provided, for example, video-based Multimodal EmotionLines Dataset (MELD). However, only a few research approaches use both acoustic and visual information MELD videos. There are two reasons this: First, label-to-video alignments noisy, making those videos an unreliable source emotional speech data. Second, can involve several people same scene, which requires localisation utterance source. In this paper, we introduce with Fixed Audiovisual Information via Realignment (MELD-FAIR) by using recent active speaker detection automatic models, able to realign capture facial expressions speakers 96.92% utterances provided MELD. Experiments self-supervised voice model indicate that realigned MELD-FAIR more closely match transcribed given dataset. Finally, devise trained on videos, outperforms state-of-the-art models ERC based vision alone. This indicates localising speaking activities is indeed effective extracting uttering faces provide informative cues than features have been so far. realignment data, code procedure recognition, available at https://github.com/knowledgetechnologyuhh/MELD-FAIR.